Power reduction for image sensor with raw image scaler

ABSTRACT

A system and method for scaling an image includes receiving raw image data comprising input pixel values which correspond to pixels of an image sensor, filtering pixels in a spatial domain, and filtering pixels in a frequency domain according to an oversampling ratio. The system and method may also include outputting scaled image data as output pixel values, which correspond to subgroups of the input pixel values. The filtering may be done according to a Bayer-consistent ruleset which includes a set of filter weights and a series of scaling rules. The oversampling ratio is set to minimize an error after the filtering in the spatial domain and the filtering in the frequency domain.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This application relates generally to digital image scaling. Morespecifically, this application relates to a raw domain image scalingsystem and method that maintains consistency while having reduced powerconsumption.

2. Description of Related Art

In a digital image capturing system, it is common that the resolutionrequirement for still image capture is higher than that of video output.In such a case, a high-resolution camera capable of supporting theresolution requirement of still image capture is often used. When thecamera is used in video mode to produce a video output stream, the datais scaled down to suit the resolution of the video output. For example,a particular image sensor with 8 million pixels (megapixels or MP) cangive 8 MP still images. For the same image sensor in video mode, aresolution of 1080×1920 pixels, or approximately 2 MP, may suffice toproduce 1080p high-definition (HD) video. Therefore, in 1080 p videomode, the image data is scaled down 3× both vertically and horizontallyto provide the desired output resolution.

This scaling may be performed either in the raw domain or the RGBdomain. An advantage of scaling the image data in the raw domain to thedesired video resolution is that it reduces the number of pixels thatmust be processed through the system. As a result, a majority ofprocessing blocks in an image pipeline or post-processing section can beoperated at a lower clock rate than the clock rate required to supportfull-resolution processing. Operating the processing section at a lowerclock rate has significant advantages in reducing electromagneticinterference and reducing power consumption of the system. Theseadvantages are especially valuable in such applications as mobileimaging.

However, existing methods of scaling in the raw domain suffer fromseveral disadvantages, including difficulty in maintaining a Bayeroutput pattern without resorting to increasingly complex, expensive, andresource-intensive logic circuitry. Additionally, such existing scalingmethods suffer from inferior image quality when compared to scaling inthe RGB domain. Moreover, to the extent that existing scaling methodsrely on such complex, expensive, and resource-intensive logic circuitry,such methods consume prohibitive amounts of power.

Accordingly, there is a need for raw image scaling that can efficientlyproduce output images of high image quality (that is, with goodresolution) which are free of the image artifacts produced by existingraw image scaling, and which are effective without consuming an undueamount of power.

BRIEF SUMMARY OF THE INVENTION

Various aspects of the present disclosure relate to a system and methodfor scaling an image. The scaling includes receiving a raw image datacomprising a plurality of input pixel values, respective ones of theinput pixel values corresponding to a respective pixel of an imagesensor; filtering the plurality of input pixel values in a spatialdomain; and filtering the plurality of input pixel values in a frequencydomain according to a predetermined oversampling ratio.

In one example, the predetermined oversampling ratio is set such that anerror after the filtering in the spatial domain and the filtering in thefrequency domain is less than a threshold value. In an additional oralternative example, the predetermined oversampling ratio isproportional to an error in an A/D conversion.

In this manner, various aspects of the present disclosure provide forimprovements in at least the underlying technical processes of imagecapturing and image processing.

This disclosure can be embodied in various forms, including businessprocesses, computer-implemented methods, computer program products,computer systems and networks, user interfaces, application programminginterfaces, hardware-implemented methods, signal processing circuits,image sensor circuits, application specific integrated circuits, fieldprogrammable gate arrays, and the like. The foregoing summary isintended solely to give a general idea of various aspects of the presentdisclosure, and does not limit the scope of the disclosure in any way.

DESCRIPTION OF THE DRAWINGS

These and other more detailed and specific features of variousembodiments are more fully disclosed in the following description,reference being had to the accompanying drawings, in which:

FIGS. 1A, 1B, and 1C respectively illustrate three different exemplaryimage capturing systems with image scaling according to various aspectsof the present disclosure.

FIGS. 2A and 2B respectively illustrate an exemplary block diagram of animage sensor according to various aspects of the present disclosure.

FIG. 3 illustrates an exemplary Bayer filter for use with variousaspects of the present disclosure.

FIG. 4 illustrates an exemplary (½)× scaling operation according tovarious aspects of the present disclosure.

FIG. 5 illustrates an exemplary filtering representation for scalingaccording to various aspects of the present disclosure.

FIG. 6 illustrates an exemplary (⅓)× scaling operation according tovarious aspects of the present disclosure.

FIG. 7 illustrates an exemplary (¼)× scaling operation according tovarious aspects of the present disclosure.

FIG. 8 illustrates an exemplary set of spatial filter coefficient arraysfor use with various aspects of the present disclosure.

FIG. 9 illustrates an exemplary set of optimized spatial filtercoefficient arrays for use with various aspects of the presentdisclosure.

FIG. 10 illustrates an exemplary effect of an edge in an input raw imageaccording to various aspects of the present disclosure.

FIG. 11 illustrates an exemplary (½)× scaling operation according tovarious aspects of the present disclosure.

FIG. 12 illustrates an exemplary (¼)× scaling operation according tovarious aspects of the present disclosure.

FIG. 13 illustrates an exemplary error curve for one sigma-deltaconversion as a function of oversampling ratio according to variousaspects of the present disclosure.

FIG. 14 illustrates an exemplary binning base line filter according tovarious aspects of the present disclosure.

FIG. 15 illustrates an exemplary horizontal-vertical binning filteraccording to various aspects of the present disclosure.

FIG. 16 illustrates an exemplary set of frequency filter componentarrays for use with various aspects of the present disclosure.

FIG. 17 illustrates errors for each pixel after exemplary ADCFtransformations according to various aspects of the present disclosure.

DETAILED DESCRIPTION

In the following description, numerous details are set forth, such asflowcharts, data tables, and system configurations. It will be readilyapparent to one skilled in the art that these specific details aremerely exemplary and not intended to limit the scope of thisapplication.

While the following description presents image scaling in the context ofa Bayer-consistent scaling, the present disclosure is not so limited andmay be similarly applied to any type of scaling as a particularapplication may require.

[Imaging System]

FIGS. 1A-C illustrate three different exemplary image capturing systems1 a-c that support both still image capture and video output, either atfull sensor resolution or at a desired image or video resolution. Allthree exemplary image capturing systems 1 a-c include an image sensor100 a-c and an image pipeline 200 a-c. FIG. 1A illustrates a system 1 awhere scaling is performed in the RGB domain within image pipeline 200a; FIG. 1B illustrates a system 1 b where scaling is performed in theraw domain within image pipeline 200 b; and FIG. 1C illustrates a system1 c where scaling is performed in the raw domain within image sensor 100c. As used herein, “scaling” means changing the resolution of the imagedata to the desired output video resolution.

In all three of FIGS. 1A-C, image pipeline 200 a-c includes a raw domainprocessing section (for example, a raw domain processing circuit) 201a-c which comprises a series of raw domain processing blocks P1, . . . ,Pn. These raw domain processing blocks process raw image data (alsocalled raw pixel data) in the same format as they are captured from theimage sensor. Additionally, image pipeline 200 a-c includes a demosaicsection (for example, a demosaic circuit) 202 a-c which converts rawimage data into full color image data; that is, data wherein each pixelincludes all RGB values. Furthermore, image pipeline 200 a-c includes anRGB domain processing section (for example, an RGB domain processingcircuit) 203 a-c which comprises a series of RGB domain processingblocks Q1, . . . , Qk. Examples of processing blocks or circuits inimage pipeline 200 a-c include lens artifact correction, bad pixelcorrection, color correction, noise filtering, sharpening, and the like.

In FIG. 1A, RGB domain processing section 203 a includes an RGB scalingsection (for example, a scaling circuit) 300 a. In FIG. 1B, raw domainprocessing section 201 b includes a raw scaling section 300 b. In FIG.1C, image sensor 100 c includes scaling section 300 c. While the threeexemplary views show only a single RGB or raw scaling section, exemplaryimage capturing systems may have the capability of supporting both rawdomain image scaling and RGB domain image scaling. In such case, both araw scaling section and an RGB scaling section would be present in thesame image capturing system. Such an arrangement would provide the useror system designer with the capability of selecting either raw scaling,RGB scaling, or both, to suit specific operating conditions as desired.

The raw scaling techniques described herein improve the functioning ofthe image sensor and/or the image pipeline by allowing it to produceoutput images of high quality with fewer artifacts and at a reducedpower consumption.

Image pipeline 200 a-c may be implemented either in hardware, software,or a mixture of both. Examples of hardware implementations includeapplication specific integrated circuit (ASIC), field programming logicarray (FPGA), other programmable logic circuits, discrete circuitelements, and the like. Examples of software implementations includefirmware in an embedded chip, software in digital signal processors(DSP), software in a simulator, software in a graphics processing unit(GPU), software in a general purpose central processing unit (CPU), andthe like. A mixture of hardware and software may also be used whereinsome blocks in image pipeline 200 a-c are implemented in hardware, withthe remaining blocks implemented in software. In one example, one ormore of image sensor 100 a-c and image pipeline 200 a-c, or subunitsthereof, are implemented as a processing unit and a memory.

FIGS. 2A-B illustrate simplified block diagrams of exemplary imagesensors 100 a-c. FIG. 2A is an example of an image sensor 100 a thatdoes not itself perform scaling; for example, the image sensor 100 a ofFIG. 1A. FIG. 2B, on the other hand, is an example of an image sensor100 c that does perform scaling, as will be described in more detailbelow.

In FIG. 2A, image sensor 100 a includes a pixel array 101, including aplurality of pixels 111 arranged in rows and columns. Individual pixels111 include pixel elements such as a photosensitive element (forexample, a photodiode) and associated control circuits (for example,transistors). The pixel array 101 is connected to a readout circuit 102so that analog data values from pixels 111 may be sent to a bank ofanalog-to-digital converters (ADCs) 103, which convert the pixel datavalues into digital form. Pixel array 101, readout circuit 102, and ADCs103 are controlled by a sequencer and timing control circuit 104 whichcontrols the ordering of pixel reading operations and the timing of thecircuit elements in pixel array 101. ADCs 103 may include single slopeADCs, cyclic ADCs, successive approximation ADCs, sigma-delta ADCs (alsoknown as delta-sigma ADCs), and the like.

In FIG. 2B, ADCs 103 are sigma-delta ADCs. A sigma-delta ADC requiresoversampling and gives out a sequence of D bit samples, where D istypically a low number such as 1. In order to generate multi-bit outputdigital values (for example, V-bit samples where V>D), a decimationfilter is required which removes high frequency content in the D-bitsample sequence. Specifically, as illustrated in FIG. 2B, sigma-deltaADC 103 is operatively connected to an ADC filter block (ADCF) 105 tooperate as a decimation filter. Also included in the sensor is a rawimage scaling block (RISF) 106 which operates to scale the raw data to adifferent output size. As will be discussed in more detail below, theraw domain image scaling calculations in RISF 106 can be implementedwith ADCF 105 as a single digital filter, which leads to a veryefficient implementation that has several advantages.

In FIG. 2B, ADCF 105 and RISF 106 are illustrated as being downstreamfrom the ADC 103. In this manner, raw scaling in the digital domain maybe performed. Alternatively or additionally, the RISF 106 may be placedupstream from ADC 103 so as to perform raw scaling in the analog domain.After scaling, the analog signal is converted by the ADC and then passthrough ADCF to produce the output.

To provide for color images, a color filter array (CFA) is provided withimage sensor 100, so that each pixel gives a data value corresponding toa single primary color. FIG. 3 illustrates a so-called “Bayer” CFA 101for use with image sensor 100. In this example, R represents the pixelscoated or provided with optical red filters, which therefore give onlyred pixel values. Similarly, B represents the pixels coated or providedwith optical blue filters, which therefore give only blue pixel values.Both G1 and G2 represent pixels coated or provided with green opticalfilters, which therefore give only green pixel values. Because the greenpixels sharing a row with red pixels and the green pixels sharing a rowwith blue pixels may have different pixel characteristics due tocross-talk and other reasons, they are labeled with the differentnotations G1 and G2, respectively. While the Bayer CFA illustrated inFIG. 3 uses an RGB layout, the CFA is not particularly limited in thisregard. For example, other color arrangements are possible such asred/green/blue/white (RGBW); cyan/magenta/yellow (CMY);cyan/magenta/yellow/green (CMYG); and the like.

In FIG. 3, RGB Bayer CFA 101 comprises a tiled or repeating arrangementof 2×2 blocks 121 of pixels 111, comprising an R pixel in the upper leftcorner thereof, a G1 pixel in the upper right corner thereof, a G2 pixelin the lower left corner thereof, and a B pixel in the lower rightcorner thereof. One of ordinary skill in the art will readily recognizethat other orderings are possible, such as B/G2/G1/R, G1/R/B/G2,G2/B/R/G1, and the like, as well as other primary or complementary colorcombinations such as C/M/Y/Y, or combinations of color and white such asR/G/B/W.

[General Scaling]

In practical imaging system implementations, the raw domain scalingmethod can be implemented either in hardware or software, or a mixtureof both. For software implementations, the calculations in the scalingmethod can be implemented using embedded processors, digital signalprocessors, general purpose processors, software simulation units, andthe like. For hardware implementations, the calculations in the scalingmethod can be implemented using digital means or analog means. Digitalimplementation in hardware uses digital logic elements such as gates,latches, arithmetic units, and the like. The logic elements can beincluded into an ASIC, an FPGA, discrete elements, or other programmablecircuits. Analog implementation in hardware can include capacitive orresistive circuit elements such as summing junctions, voltage or currentdividers, operational amplifiers, and the like.

The raw scaling method can be considered a filtering process followed bydecimation. Scaling occurs according to a scaling factor (1/N)×, where aBayer input region of 2N×2N is processed to produce a 2×2 Bayer outputregion.

FIG. 4 illustrates a general example of a scaling method where N=2; thatis, having a scaling factor of (½)×. That is, the output data has aresolution in both the row and column directions which is ½ theresolution of the input data. In this example, a Bayer input region of4×4 is processed to produce a 2×2 Bayer output region. As illustrated inFIG. 4, input raw data 401 is converted into scaled output data 402 byapplying a scaling transformation 403, which is illustrated in moredetail in FIG. 5.

In FIG. 5, raw input pixels in each tile of size 4×4 are processed byfour filters to give a 2×2 block in the output raw image. In theparticularly illustrated filtering procedure, the pixel values of eachred pixel are considered to be in a red pixel array R defined as{r_(1,1), r_(1,2), r_(2,1), r_(2,2)}. Red pixel array R is operated onby a red spatial filter coefficient array H defined as {h0, h1, h2, h3}to provide an output red pixel R′. Specifically, each element of redpixel array R is multiplied by the corresponding element of red spatialfilter coefficient array H. That is, the coefficients in the red filterarray are weights to be applied to the corresponding pixel values in thered pixel array. The products are then added together and divided by thesum of all elements of red spatial filter coefficient array H to producethe output red pixel R′. Mathematically, this is represented by thefollowing expression (1):

$\begin{matrix}{R^{\prime} = \frac{{r_{1,1}h\; 0} + {r_{1,2}h\; 1} + {r_{{2,1}\;}h\; 2} + {r_{2,2}h\; 3}}{{h\; 0} + {h\; 1} + {h\; 2} + {h\; 3}}} & (1)\end{matrix}$

Similarly, the pixel values of each green pixel in red rows areconsidered to be in a first green pixel array G1 defined as {g1_(1,1),g1_(1,2), g1_(2,1), g1_(2,2)} and are operated on by a first greenspatial filter coefficient array I defined as {i0, i1, i2, i3} to outputfirst green pixel G1′; the pixel values of each green pixel in blue rowsare considered to be in a second green pixel array G2 defined as{g2_(1,1), g2_(1,2), g2_(2,1), g2_(2,2)} and are operated on by a secondgreen spatial filter coefficient array J defined as {j0, j1, j2, j3} tooutput second green pixel G2′; and the pixel values of each blue pixelare considered to be in a blue pixel array B defined as {b_(1,1),b_(1,2), b_(2,1), b_(2,2)} and are operated on by a blue spatial filtercoefficient array K defined as {k0, k1, k2, k3} to output blue pixel B′.These operations are represented by the following expressions (2)-(4):

$\begin{matrix}{{G\; 1^{\prime}} = \frac{{g\; 1_{1,1}i\; 0} + {g\; 1_{1,2}i\; 1} + {g\; 1_{2,1}i\; 2} + {g\; 1_{2,2}h\; 3}}{{i\; 0} + {i\; 1} + {i\; 2} + {i\; 3}}} & (2) \\{{G\; 2^{\prime}} = \frac{{g\; 2_{1,1}j\; 0} + {g\; 2_{1,2}j\; 1} + {g\; 2_{2,1}j\; 2} + {g\; 2_{2,3}j\; 3}}{{j\; 0} + {j\; 1} + {j\; 2} + {j\; 3}}} & (3) \\{B^{\prime} = \frac{{b_{1,1}k\; 0} + {b_{1,2}k\; 1} + {b_{2,1}k\; 2} + {b_{2,2}k\; 3}}{{k\; 0} + {k\; 1} + {k\; 2} + {k\; 3}}} & (4)\end{matrix}$

The filtering representation described above can be generalized to otherscaling factors (1/N)× for any integer N. For a scaling factor (1/N)×,an input region of size 2N×2N is used. In this manner, to produce anoutput pixel of a particular color, only input pixels of the same colorare considered as an input to the filter. As a result, each filter arrayincludes only N² coefficients. Alternatively, it is possible tocalculate a particular color output pixel value using the pixels ofother colors. In that case, the number of spatial filter coefficientsused for a scaling factor (1/N)× will have more than N² terms.Additionally, the regions of support of the filters for a scaling factor(1/N)× can be expanded beyond the 2N×2N region, and in such a case thenumber of spatial filter coefficients used for a scaling factor (1/N)×will also have more than N² terms.

[Pixel Skipping and Binning]

Conceptually, a pixel skipping method at a scaling factor (1/N)× is onethat tiles an original image with 2N×2N cells, and for each cell retainsonly the four pixels in a 2×2 configuration in the upper left corner ofeach cell. All other pixels in the cell are discarded or skipped.

Using the above filtering representation, raw pixel scaling using thepixel skipping method may be represented by the filters H={1, 0, 0, 0};I={1, 0, 0, 0}; J={1, 0, 0, 0}; and K={1, 0, 0, 0}. For convenience ofnotation in the spatial filter coefficient arrays, only the coefficientscorresponding to pixels of the same color are written.

The pixel skipping method may lead to a loss of information because manypixels are simply ignored. That is, for a scaling factor of (1/N)×, onlyone out of every N² pixels are retained and the rest are discarded.

Conceptually, a binning method at a scaling factor (1/N)× is one thattiles an input raw image with 2N×2N cells, and for each cell calculatesthe arithmetic averages of each color as the respective pixel values ina 2×2 cell of the output raw image. For purposes of this calculation, G1and G2 are treated as different colors, even though they both correspondto the color green, and the averages for G1 and G2 are calculatedindependently.

Again using the above filtering representation, raw pixel scaling usingthe binning method may be represented by the filters H={¼, ¼, ¼, ¼};I={¼, ¼, ¼, ¼}; J={¼, ¼, ¼, ¼}; and K={¼, ¼, ¼, ¼}. Again, forconvenience of notation, only the coefficients corresponding to pixelsof the same color are written.

In contrast to pixel skipping, binning represents an opposite approachwhere all the N² pixels are retained with equal weights in the outputimage. This may lead to a loss of resolution, create aliasing artifacts,result in uneven phase in pixels of different colors, and the like.

[Bayer-Consistent Scaling]

Pixel skipping and binning methods typically produce output images ofsub-optimal quality; for example, having image artifacts, requiringexpensive correction circuits, and the like. As a result, there is aneed for a raw domain image scaling method which maintains Bayerconsistency.

A raw domain image scaling method that maintains Bayer consistency iscalled “Bayer-consistent raw scaling” (BCRS). An example of BCRS with ascaling factor of (⅓)× (that is, N=3) is illustrated in FIG. 6. In thiscase, the input raw image is divided into tiles of size 6×6, and fromeach 6×6 input tile the scaling method calculates a 2×2 block of theoutput image. As illustrated, each 6×6 input tile 610 is divided intofour 3×3 sub-tiles 611-614 that respectively represent the physical areaof virtual pixel 621-624 in a block 620 of a virtual image sensor havinga lower resolution equal to the scaled resolution. That is, the virtualimage sensor would have ⅓ the resolution both vertically andhorizontally compared to the actual image sensor used for capturing theraw image data.

Sub-tiles 611-614 are identified by the color of the correspondingvirtual pixel 621-624, under the assumption that the virtual imagesensor uses the same Bayer CFA as the actual image sensor. For example,sub-tile 611 is identified as an R sub-tile because it corresponds tovirtual pixel 621, which is positioned where the R filter would be in aBayer CFA on the virtual image sensor. Similarly, sub-tile 612 isidentified as a G1 sub-tile, sub-tile 613 as a G2 sub-tile, and 614 as aB sub-tile.

A similar example of BCRS with a scaling factor of (¼)× is illustratedin FIG. 7. That is, the input raw image is divided into tiles of 8×8,and from each 8×8 input tile the scaling method calculates a 2×2 outputblock. Here, each 8×8 tile 710 is divided into four 4×4 sub-tiles711-714 that respectively represent the physical area of a virtual pixel721-724 in a block 720 of a virtual image sensor having a resolutionequal to ¼ the resolution both vertically and horizontally compared tothe actual image sensor.

In both FIGS. 6-7, the geometric pixel centers of the virtual pixels621-624 and 721-724 are indicated by dashed lines. The correspondingphysical locations are indicated by arrows in input tiles 610 and 710;these are collocated with the centers of the sub-tiles 611-614 and711-714.

In light of the above requirements, BCRS is performed so as to satisfythe following conditions: (1) only input pixels of the same color as theoutput are used in the calculation of the output pixel value; (2) thefilter weights are concentrated within the sub-tile of the same color;and (3) the center of gravity of the filter weights for each colorcoincides with the geometric pixel center of the output pixel of thesame color. Taken together, these Bayer-consistency conditions form aBayer-consistency ruleset, which provide for improved scaling quality.

To measure the degree of Bayer consistency, a criterion called aBayer-consistency coefficient C may be defined which includes twocomponents corresponding to conditions (2) and (3) above. The firstcomponent γ measures the concentration of filter weights within thesub-tile of the same color, and is defined according to the followingexpressions (5) and (6):

$\begin{matrix}{\gamma = {\frac{1}{4}{\sum\limits_{c}\gamma_{c}}}} & (5) \\{\gamma_{c} = \frac{\sum_{c\mspace{14mu} {sub}\text{-}{tile}}( {{filter}\mspace{14mu} {coefficients}\mspace{14mu} {for}\mspace{14mu} c\mspace{14mu} {colored}\mspace{14mu} {pixels}} )^{2}}{\sum_{{all}\mspace{14mu} {sub}\text{-}{tiles}}( {{filter}\mspace{14mu} {coefficients}\mspace{14mu} {for}\mspace{14mu} c\mspace{14mu} {colored}\mspace{14mu} {pixels}} )^{2}}} & (6)\end{matrix}$

For a filter of color c where the weights are completely concentratedwithin the c sub-tile (that is, with non-zero weights only inside the csub-tile), γ_(c) achieves a maximum value of 1. Accordingly, each γ_(c)has a possible range of values between 0 and 1, and hence γ is alsobetween 0 and 1.

The second component β measures the deviation of the center of gravitiesof the filter weights from the geometric pixel centers of the outputimage; that is, geometric pixel centers of a lower resolution virtualgrid. It is defined according to the following expressions (7) and (8):

$\begin{matrix}{\beta = {\max( {{1 - {\frac{1}{4}{\sum\limits_{c}D_{c}}}},0} )}} & (7) \\{D_{c} = \frac{\sqrt{( d_{c}^{h} )^{2} + ( d_{c}^{v} )^{2}}}{N\; \Delta}} & (8)\end{matrix}$

Above, Δ is the length of the input pixel, d_(c) ^(h) and d_(c) ^(v) arethe horizontal and vertical distances, respectively, between the centerof gravity of the spatial filter coefficients for the color c and thegeometric pixel center of the c color pixel in the corresponding virtualgrid, and N is the integer in the scaling factor (1/N)×. For a filterwhere the center of gravities of the spatial filter coefficient in allfour colors coincide with the geometric pixel center of the output pixelof respective colors, D_(c)=0 for all c and β achieves a maximum valueof 1. Hence, β is between 0 and 1.

The calculations of D_(c) in expression (8) above assume that the pixelshave a square shape; that is, both the width and height of the pixelsare equal to Δ. In the case of rectangular pixels, the calculations canbe performed by first normalizing d_(c) ^(h) and d_(c) ^(v) by the widthand height, respectively, of the pixel, and then calculating D_(c) asthe root mean square of the normalized values.

To consider the overall effect of both γ and β, Bayer-consistencycoefficient C is defined according to the following expression (9):

C=μγ+(1−μ)β  (9)

The parameter μ has a value between 0 and 1, and is used as a weight forthe two components γ and β. Therefore, as is readily apparent, C has apossible range of values between 0 and 1. For performance evaluation,μ=0.5 may be used. The performance evaluation parameter μ has a valuebetween 0 and 1, and is used as a weight for the two components γ and β.In the above expression, a high γ indicates a high degree of imagesharpness, whereas a high β indicates an image free from jagged edges. Avalue of μ=0.5 is chosen for a balance between image sharpness and animage free of jagged edges. Using this definition, raw scaling filterswith higher values of C are preferred. In other words, filters whichexhibit a higher degree of Bayer-consistency lead to higher imagequality; for example, having C≧0.65. More preferably, C≧0.8. Mostpreferably, C>0.9.

For comparison, consider the pixel skipping and binning scaling methodsdescribed above. For pixel skipping at a scaling factor of (½)×, thevalues of γ_(c) are 1, 0, 0, 0 for the colors R, G1, G2, and B,respectively. Therefore, γ=0.25. Additionally, it can be calculated thatthe horizontal distance d_(c) ^(h) equals 0.5Δ, 1.5Δ, 0.5Δ, and 1.5Δ forR, G1, G2, and B, respectively; whereas the vertical distance d_(c) ^(v)equals 0.5Δ, 0.5Δ, 1.5Δ, and 1.5Δ for R, G1, G2, and B, respectively.Accordingly, β=0.2512. Using μ=0.5 for evaluation, the Bayer consistencycoefficient C for pixel skipping is 0.2506 for a scaling factor of (½)×.By similar calculations, the Bayer consistency coefficient C for pixelskipping equals 0.1258 for (⅓)× and 0.1250 for (¼)×.

For binning at a scaling factor of (½)×, γ_(c) is(¼²)/(¼²+¼²+¼²+¼²)—that is, 0.25—for each color c, and therefore γ=0.25.Both the horizontal distance d_(c) ^(h) and the vertical distance d_(c)^(v) equal 0.5Δ for any color c, and therefore β=0.6464. Again usingμ=0.5 for evaluation, the Bayer consistency coefficient C for binning is0.4482 for a scaling factor of (½)×. By similar calculations, the Bayerconsistency coefficient C for binning equals 0.4865 for (⅓)× and 0.3598for (¼)×.

On the other hand, a BCRS filter configured to satisfy the three Bayerconsistency conditions defined above provides a high Bayer consistencycoefficient value and good image quality. FIG. 8 provides an example ofsuch a filter for a scaling factor of (⅓)×. For the particular filterillustrated in FIG. 8, γ_(c) is (¼²+¼²+¼²+¼²)/(¼²+¼²+¼²+¼²) for any c.Therefore, γ_(c) is 1 for any color c. Additionally, D_(c)=0 for anycolor c. As a result, C achieves its highest possible value of 1. Anextension to general scaling factors (1/N)× for any integer N whichachieves the maximum value of C=1 is straightforward.

[Optimized Bayer-Consistent Scaling]

Pure BCRS can potentially produce some false colors in the highfrequency areas of the output images, which can be observed fromprocessing images of resolution charts. This is due to the maintenanceof very high resolution in the scaled output images. To make furtherimprovements, the spatial filter coefficients may be optimized. Theoptimization procedure for each color c involves allowing somecoefficients outside of the c sub-tile to take on a non-zero value whichis substantially smaller than the coefficient values inside the csub-tile, and in the process evaluating the resulting Bayer consistencycoefficient C and false coloring in the output image. While thisoptimization procedure implies that condition (2) described above nolonger holds in a strict sense, the overall Bayer consistencycoefficient C is still evaluated in the optimization to ensure a high Cvalue so that a majority of the weights of the filter remains inside thesub-tile of the same color.

FIG. 9 illustrates an example of an optimized filter for a scalingfactor of (⅓)×. For the filters in FIG. 9, γ, d_(c) ^(h), and d_(c) ^(v)are determined in the following expressions (10) and (11):

$\begin{matrix}{\gamma = {\gamma_{c} = {\frac{4 \times ( \frac{13}{64} )^{2}}{{4 \times ( \frac{13}{64} )^{2}} + {2 \times ( \frac{3}{64} )^{2}} + {3 \times ( \frac{2}{64} )^{2}}} = 0.9575}}} & (10) \\{d_{c}^{h} = {d_{c}^{v} = {\frac{{( {- 1} )( {{2 \times 13} + 3} )} + ( {{2 \times 13} + 2} ) + {(3)( {3 + {2 \times 2}} )}}{64} = 0.3125}}} & (11)\end{matrix}$

The above values hold for any color c, and therefore β=0.8527.Evaluating at μ=0.5, these components lead to a Bayer consistencycoefficient of 0.9051, which is both close to the maximum possible valueof 1 and much higher than the corresponding values for pixel skipping(0.1258) and binning (0.4865) for a scaling factor (⅓)×.

BCRS is especially effective because the Bayer consistency conditionsrequire that the structural information (for example, edges) of thecaptured images be localized and kept at the correct locations in thescaled image. This is illustrated by FIG. 10. Specifically, FIG. 10shows an edge 1001 in the scene appearing in the lower right portion ofthe pixel area. If an image including edge 1001 were to be scaled downby the binning method, the effect of the edge would affect all fouroutput pixels because pixels within the edge area are used incalculating the output raw pixels of all colors. In this case the outputR, G1, and G2 pixels, even though located outside of the area containingedge 1001, are affected by the presence of edge 1001. This means thatthe edge 1001 is spread out into the entire 2×2 output area, whichcorresponds to the entire 6×6 input area. On the other hand, by usingBCRS (for example, using the filters of FIG. 8), only the R pixels inthe R sub-tile are used in the calculation of the output R pixel value.Therefore, the output R value is not affected by the presence of edge1001. Similarly, the output G1 and G2 values are not affected by edge1001. That is, because edge 1001 is confined to the B sub-tile, only theoutput B value is affected by edge 1001. In this manner, BCRS preservesthe structure of the image more accurately and produces output images ofhigh resolution.

Although, in optimized BCRS (for example, using the filters of FIG. 9),pixels outside of the c sub-tile are used in the calculation of theoutput c-colored pixel, the structural limitation is similarly preservedbecause the filter weights of the pixels outside of the same colorsub-tile are comparatively low.

Optimized BCRS filters may also be used to scale at scaling factorsother than (⅓)×; for example, any (1/N)× for integer N. For example,FIG. 11 illustrates an exemplary optimized BCRS red spatial filtercoefficient array for scaling factor (½)×; whereas FIG. 12 illustratesan exemplary optimized BCRS red spatial filter coefficient array forscaling factor (¼)×. Using the above expressions at μ=0.5, the BCRSfilter arrays of FIGS. 11 and 12 give a Bayer consistency coefficient Cof 0.8279 and 0.8166, respectively. These are much higher than the Bayerconsistency coefficients C for both pixel skipping and binning asdescribed above.

[Oversampling]

Generally, the power consumption of an image sensor ADC is dependent onthe level of data accuracy required for the converted output data. Inthe case of a sigma-delta ADC, data accuracy is equivalent to the RMSnoise, and thus a function of the oversampling ratio (OSR). Here, theOSR is a predetermined ratio. While increasing the OSR leads to a higherlevel of data accuracy, this is accompanied by an increase in powerconsumption.

It is possible to reduce the signal-to-noise ratio (SNR) of a singleconverted pixel without affecting the SNR of the scaled pixel. Forexample, in the case where a scaling factor of 2 is the x-direction isperformed, the filter {½, ½} may be used. The SNR of a pixel after A/Dconversion may be defined as SNR0. With this filter, the SNR of thescaled data will be SNR0×√2. That is, the scaled pixel SNR is higherthan the SNR of the single converted pixel SNR0. This increased SNR isnot strictly required and thus the single pixel conversion accuracy maybe reduced without negatively affecting the end result of the scalingoperation. In this example, the SNR may be reduced by a factor of √2such that the SNR of the scaled data is simply SNR0.

This may be extended to situations where the SNR for each convertedpixel is not necessarily equal; that is, the case where the scalingfilter has different spatially distributed weights for each pixel. Forexample, the scaling filter {¼, ½, ¼)} results in the SNR of the scaledpixel of SNR0× √(8/3), which is again higher than SNR0. Here, then, theSNR may be reduced by this factor such that the SNR of the scaled datais SNR0. This renormalization may be performed prior; for example, thecentral pixel (coefficient=½) may be treated as having SNR1=SNR0 whereasthe two border pixels (coefficient=¼) may be treated as havingSNR2=SNR3=SNR0/(√6). In any event, after the scaling operation the SNRof the converted pixel will have the same value of SNR0.

The desired SNR for each single converted pixel may be selected from anumber of options to achieve the required scaled pixel SNR. Thisflexibility may be used to optimize ADC power consumption, as will bedetailed below.

FIG. 13 illustrates the error in 14 bits for the sigma-delta conversionas a function of the number of filter coefficients used in the ADCdigital filter, which corresponds to the OSR. While the exact numbersfor both the x- and y-axes correspond to a specific implementation, thegeneral relationship is not limited to one specific implementation. Inthe general case, the relationship is nonlinear with the conversionerror decreasing quickly with an increase in the OSR. Specific rate ofdecrease in noise depends on the ADC filter, the structure (order) ofthe sigma-delta modulator, and other factors. These relationships arewell known in the art and hence will not be further described.

In one aspect of the present disclosure, a noise target for the outputpixels (either single or after binning) is selected for singleconversion with 160 ADCF coefficients (that is, OSR=160). In this case,the ADCF error is close to the 12-bit quantization error and equal to0.31 DN12 where DN12 is digital number at 12 bits per pixel (bpp), i.e.the size of one least significant bit in 12 bpp.

To illustrate this filtering, the case of (⅓)× (that is, N=3) isimplemented using five different binning filter approaches. These may begeneralized to extend to cases of (1/N) scaling in a straightforwardmanner. These approaches are:

(1) Binning base line filter that uses pixel binning in the horizontaldirection and pixel skipping in the vertical direction, as illustratedin FIG. 14. Each single pixel is converted with an OSR=160.

(2) Pixel binning in both the horizontal and vertical directions, asillustrated in FIG. 15. As above, each single pixel is converted withOSR=160.

(3) Scaling using base BCRS filter of the type described above, asillustrated in FIG. 8. As above, each single pixel is converted withOSR=160.

(4) Scaling using base BCRS filter of the type described above, asillustrated in FIG. 8, and further with OSR optimization. Here, the OSRfor each pixel conversion is selected to be 80.

(5) Scaling using full BCRS with OSR optimization using a filter withcoefficients as illustrated in FIG. 16. Here the OSR for each pixelconversion is selected to be different and varies from 28 to 72. Thetotal number of OSR coefficients is equal to 440.

In evaluating the noise level, the quantization error after eachmathematical operation must be taken into account. This error is definedby the bit depth of each converted pixel after the ADC conversion andthe output bit depth after the scaling procedure. The standard deviationof this error equals the least significant bit (LSB)×√( 1/12).

Where the flow has 12-bit data depth after the ADC conversion and 12-bitdepth after the binning procedure, the error for the single ADCconversion is represented by the following expression (12):

$\begin{matrix}{E = \sqrt{E_{SD}^{2} + \frac{1}{12}}} & (12)\end{matrix}$

Above, E represents the error, and E_(SD) represents the sigma-deltaerror. In a similar manner, the error after the binning procedure isrepresented by the following expression (13):

$\begin{matrix}{E_{Bin} = \sqrt{{\sum\limits_{i = 1}^{M}{a_{i}^{2}\frac{E_{i}^{2}}{( {\sum_{j = 1}^{M}a_{j}} )^{2}}}} + \frac{1}{12}}} & (13)\end{matrix}$

Above, a_(i) is a weight coefficient of the scaling filter for the pixeli, and E_(i) is the error of the single sigma-delta transformationwithout added quantization error for the pixel i. Note that equation(12) describes the error for a single pixel, whereas equation (13)describes the error for an output pixel after scaling which iscalculated from M pixels (e.g. binning, BCRS, etc). The two equationshave similar forms. The weighted averaging in (13) shows the effect offiltering on the output noise. As noted above, a different OSR may beused for each converted pixel, thus resulting in a different ADC errorfor each pixel.

Thus, using expression (12), it is possible to calculate the error foreach pixel after ADCF transformation and 12-bit data storage, and tocompare the relative errors for various filter approaches, such as thosedescribed above as approaches (1)-(5). These results are shown in FIG.17 as corresponding arrays (1)-(5). For ease of illustration, only the Rpixel errors are shown for the various approaches; the Gr, Gb, and Bpixel errors follow in a straightforward manner.

Finally, using expression (13), it is possible to calculate the error ofthe pixel after the binning procedure and storing the result as 12-bitdata for each approach. These results are summarized in Table 1 below,which shows the number of OSR coefficients used for each pixel, thetotal number of input image pixels (NIIP) used for creating one binnedpixel in addition to the total output error.

(1) (2) (3) (4) (5) OSR used for (160, 160, 160) (160, 160, 160, (160,160, (80, 80, 80) (72, 72, 34, the single 160, 160, 160, 160, 160) 72,72, 28, pixel 160, 160, 160) 34, 28, 280 transformation NIIP 480 (3 ×160) 1440 (9 × 160) 640 (4 × 160) 320 (4 × 80) 440 Error after 0.31 0.310.33 0.47 0.53 Binning DN12

In implementation of the filtering, the ADCF and RISF can be implementedeither separately or jointly. For example, consider case (5) shown inthe table. There are 9 binary bit streams from the sigma-delta modulatorcorresponding to the 9 input image pixel positions. Each binary sampleis summed using weights equal according to the product of the ADCFfilter coefficient (as a function of time) and the RISF filtercoefficient (as a function of image coordinate). A total of 440 termscorresponding to the 9 input image pixel positions as shown in thefigure are summed. For other oversampling ratios and other scalingfactors, a similar procedure can be performed.

In Table 1 above, the particular NIIP selected for approaches (4) and(5) is only one of a number of possibilities, although this particularOSR is based on theoretically preferable values. Therefore, allapproaches have an error which is less than two times the target valueof 0.31. The error value 0.31 was the result of a single conversionusing an ADCF with an OSR 160. This error is close to the theoreticallimit of quantization error, which is the square root of 1/12, or 0.29.However, the BCRS approaches with OSR optimization (4) and (5) use asmaller OSR for the single pixel conversions as compared to, forexample, approach (1). Thus, it is possible to reduce the powerconsumption requirements of BCRS without compromising image quality,with only an inconsequentially small increase in the error due to anypotential inaccuracies in the ADC conversion. Generally, the system canbe designed so that the final error after filtering is less than twotimes the sigma-delta error. For the example in Table 1, the upper errorvalue would be two times 0.31 or 0.62.

CONCLUSION

With regard to the processes, systems, methods, heuristics, etc.described herein, it should be understood that, although the steps ofsuch processes, etc. have been described as occurring according to acertain ordered sequence, such processes could be practiced with thedescribed steps performed in an order other than the order describedherein. It further should be understood that certain steps could beperformed simultaneously, that other steps could be added, or thatcertain steps described herein could be omitted. In other words, thedescriptions of processes herein are provided for the purpose ofillustrating certain embodiments, and should in no way be construed soas to limit the claims.

Accordingly, it is to be understood that the above description isintended to be illustrative and not restrictive. Many embodiments andapplications other than the examples provided would be apparent uponreading the above description. The scope should be determined, not withreference to the above description, but should instead be determinedwith reference to the appended claims, along with the full scope ofequivalents to which such claims are entitled. It is anticipated andintended that future developments will occur in the technologiesdiscussed herein, and that the disclosed systems and methods will beincorporated into such future embodiments. In sum, it should beunderstood that the application is capable of modification andvariation.

All terms used in the claims are intended to be given their broadestreasonable constructions and their ordinary meanings as understood bythose knowledgeable in the technologies described herein unless anexplicit indication to the contrary in made herein. In particular, useof the singular articles such as “a,” “the,” “said,” etc. should be readto recite one or more of the indicated elements unless a claim recitesan explicit limitation to the contrary.

The Abstract of the Disclosure is provided to allow the reader toquickly ascertain the nature of the technical disclosure. It issubmitted with the understanding that it will not be used to interpretor limit the scope or meaning of the claims. In addition, in theforegoing Detailed Description, it can be seen that various features aregrouped together in various embodiments for the purpose of streamliningthe disclosure. This method of disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter lies in less than allfeatures of a single disclosed embodiment. Thus the following claims arehereby incorporated into the Detailed Description, with each claimstanding on its own as a separately claimed subject matter.

1. A method of scaling an image in an image scaling circuit, comprising:receiving, at the image scaling circuit, a raw image data comprising aplurality of input pixel values, respective ones of the input pixelvalues corresponding to a respective pixel of an image sensor;filtering, by the image scaling circuit, the plurality of input pixelvalues in a spatial domain; filtering, by the image scaling circuit,respective ones of the plurality of input pixel values in a frequencydomain according to a corresponding predetermined oversampling ratio. 2.The method according to claim 1, wherein the corresponding predeterminedoversampling ratio is set such that an error after the filtering in thespatial domain and the filtering in the frequency domain is less thantwo times an error of a single sigma-delta conversion.
 3. The methodaccording to claim 1, further comprising: in the step of filtering inthe spatial domain, converting, by the image scaling circuit, the rawimage data from an analog data to a digital data.
 4. The methodaccording to claim 3, wherein the corresponding predeterminedoversampling ratio is proportional to an error in the step ofconverting.
 5. The method according to claim 1, further comprising: inthe step of filtering in the spatial domain, filtering, by the imagescaling circuit, the plurality of input pixel values according to aBayer-consistent ruleset.
 6. The method according to claim 5, whereinthe Bayer-consistent ruleset includes: a plurality of spatial filterweights; a first rule that a respective output pixel value correspondsto a first color, and the corresponding subgroup of the plurality ofinput pixel values also corresponds to the first color; a second rulethat the plurality of input pixel values corresponds to a plurality ofinput sub-tiles and the spatial filter weights for the first color areconcentrated within an input sub-tile of the same color; and a thirdrule that a center of gravity of the spatial filter weights for thefirst color coincides with the geometric pixel center of the outputpixel of the first color.
 7. The method according to claim 1, furthercomprising: outputting, from the image scaling circuit, scaled imagedata comprising a plurality of output pixel values, respective ones ofthe output pixel values corresponding to a subgroup of the plurality ofinput pixel values.
 8. The method according to claim 1, wherein thecorresponding predetermined oversampling ratio for a scaling factorlarger than 1 is smaller than the corresponding predeterminedoversampling ratio for a scaling factor equal to
 1. 9. An imageprocessing circuit, comprising: an image scaling circuit including: aninput unit configured to receive raw image data comprising a pluralityof input pixel values, respective ones of the input pixel valuescorresponding to a respective pixel of an image sensor; and a filterunit configured to filter respective ones of the plurality of pixels ina spatial domain, and to filter the plurality of input pixel values in afrequency domain according to a corresponding predetermined oversamplingratio.
 10. The image processing circuit according to claim 9, whereincorresponding the predetermined oversampling ratio is set such that anerror after the filtering in the spatial domain and the filtering in thefrequency domain is less than two times an error of a single sigma-deltaconversion.
 11. The image processing circuit according to claim 9,wherein the image scaling circuit further includes an analog-to-digitalconverter.
 12. The image processing circuit according to claim 11,wherein the corresponding predetermined oversampling ratio isproportional to an error in the step of converting.
 13. The imageprocessing circuit according to claim 9, wherein the filter unit isconfigured to filter the plurality of pixels in the spatial domainaccording to a Bayer-consistent ruleset.
 14. The image processingcircuit according to claim 13, wherein the Bayer-consistent rulesetincludes: a plurality of spatial filter weights; a first rule that arespective output pixel value corresponds to a first color, and thecorresponding subgroup of the plurality of input pixel values alsocorresponds to the first color; a second rule that the plurality ofinput pixel values corresponds to a plurality of input sub-tiles and thespatial filter weights for the first color are concentrated within aninput sub-tile of the same color; and a third rule that a center ofgravity of the spatial filter weights for the first color coincides withthe geometric pixel center of the output pixel of the first color. 15.The image processing circuit according to claim 9, further comprising:an output unit configured to output scaled image data comprising aplurality of output pixel values, respective ones of the output pixelvalues corresponding to a subgroup of the plurality of input pixelvalues.
 16. The image processing circuit according to claim 9, whereinthe corresponding predetermined oversampling ratio for a scaling factorlarger than 1 is smaller than the corresponding predeterminedoversampling ratio for a scaling factor equal to
 1. 17. An imagingdevice, comprising: an image sensor including a plurality of pixels; andan image scaling circuit including: an input unit configured to receiveraw image data comprising a plurality of input pixel values, respectiveones of the input pixel values corresponding to a respective pixel ofthe image sensor; and a filter unit configured to filter the pluralityof pixels in a spatial domain, and to filter respective ones of theplurality of input pixel values in a frequency domain according to acorresponding predetermined oversampling ratio.
 18. The imaging deviceaccording to claim 17, wherein the corresponding predeterminedoversampling ratio is set such that an error after the filtering in thespatial domain and the filtering in the frequency domain is less thantwo times an error of a single sigma-delta conversion.
 19. The imagingdevice according to claim 17, wherein the image scaling circuit furtherincludes an analog-to-digital converter.
 20. The imaging deviceaccording to claim 19, wherein the corresponding predeterminedoversampling ratio is proportional to an error in the step ofconverting.
 21. The imaging device according to claim 17, wherein thefilter unit is configured to filter the plurality of pixels in thespatial domain according to a Bayer-consistent ruleset.
 22. The imagingdevice according to claim 21, wherein the Bayer-consistent rulesetincludes: a plurality of spatial filter weights; a first rule that arespective output pixel value corresponds to a first color, and thecorresponding subgroup of the plurality of input pixel values alsocorresponds to the first color; a second rule that the plurality ofinput pixel values corresponds to a plurality of input sub-tiles and thespatial filter weights for the first color are concentrated within aninput sub-tile of the same color; and a third rule that a center ofgravity of the spatial filter weights for the first color coincides withthe geometric pixel center of the output pixel of the first color. 23.The imaging device according to claim 17, further comprising: an outputunit configured to output scaled image data comprising a plurality ofoutput pixel values, respective ones of the output pixel valuescorresponding to a subgroup of the plurality of input pixel values. 24.The imaging device according to claim 17, wherein the correspondingpredetermined oversampling ratio for a scaling factor larger than 1 issmaller than the corresponding predetermined oversampling ratio for ascaling factor equal to
 1. 25. The method according to claim 1, whereinrespective ones of the plurality of pixel values are filtered in thefrequency domain by corresponding predetermined oversampling ratiosdifferent from each other.
 26. The image processing circuit according toclaim 9, wherein the filter unit is configured to filter respective onesof the plurality of input pixel values in the frequency domain bycorresponding predetermined oversampling ratios different from eachother.
 27. The imaging device according to claim 17, wherein the filterunit is configured to filter respective ones of the plurality of inputpixel values in the frequency domain by corresponding predeterminedoversampling ratios different from each other.